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INTRODUCTION 


As  units  under  test  (UUT)  testable  with  automatic  test  equipment  (ATE) 
increase  in  both  number  and  complexity,  and  the  technically  experienced 
personnel  required  to  perform  independent  verification  and  validation 
(IV&V)  of  UUT  test  programs  continue  to  be  in  short  supply,  alternative 
means  of  performing  test  program  IV&V  must  be  developed  in  order  to 
reduce  the  significant  investment  in  time  and  resources  currently  devoted 
to  this  effort.  The  primary  requirement  of  the  alternative,  however,  must 
be  to  ensure  the  suitability  and  correctness  of  the  test  program  for 
detecting  and  correctly  diagnosing  UUT  failures. 

The  Expert  System  for  TPS  Fault  Candidate  Selection  is  one  means  to 
achieve  this  objective.  To  assist  in  performing  test  program  IV&V,  the 
expert  system  analyzes  characteristics  of  the  test  program  (TP)  and  UUT 
circuit  architecture,  performs  several  quality  assurance  functions  relating 
to  TP  and  UUT  consistency,  then  selects  faults  for  insertion  in  the  UUT 
based  on  test  program  structure,  circuit  characteristics  and  the  application 
of  rules  utilized  by  human  experts  in  accomplishing  this  task. 

The  expert  system  was  developed  and  successfully  demonstrated  as  a 
prototype  to  ARDEC  in  March  1987.  All  functions  required  in  the 
production  expert  system  were  fully  incorporated  into  the  prototype,  and 
three  existing  test  programs  in  ARDEC's  inventory  were  used  to  validate 
the  results  in  testing  the  expert  system.  In  subsequent  paragraphs, 
general  principles  of  fault  selection  criteria  for  TPS  IV&V  will  be 
addressed,  along  with  the  hypotheses  for  their  implementation  in  an 
expert  system  environment.  Benchmark  test  results  will  be  analyzed  for 
overall  performance  and  compared  to  results  obtained  from  human 
experts. 
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Human  Approach  to  Test  Program  Independent  Verification  and  Validation 


Independent  Verification  and  Validation  for  test  programs  is  typically 
performed  in  stages  by  QA  Engineers.  The  first  stage  encompasses  review 
of  UUT  source  data  to  gain  an  understanding  of  the  function  and  operation 
of  the  UUT.  This  is  followed  by  review  of  the  test  program  to  evaluate  the 
testing  performed  therein.  Much  of  this  effort  entails  routine  comparison 
of  the  test  program  content  to  UUT  content  to  determine: 

1.  if  all  detectable  component  failure  modes  are  called  out  in  the  program, 

2.  whether  ambiguity  group  sizes  and  fault  isolation  capability  meet 

contractual  requirements,  and 

3.  other  bookkeeping  tasks  that  need  to  be  performed. 

Once  the  engineer  is  satisfied  the  program  addresses  the  basic 
characteristics  of  the  UUT  and  the  contract.  The  test  philosophy  and  logic 
flow  are  analyzed  to  ensure  the  test  program  actually  performs  the  testing 
that  is  apparent  on  the  surface.  With  this  done,  the  engineer  selects  fault 
modes  of  components  for  inclusion  in  a  fault  list,  to  then  be  demonstrated 
as  detectable  and  correctly  diagnosed  by  the  test  program.  Fault  selection 
and  fault  list  development  is  highly  subjective.  It  is  based  primarily  on 
the  previous  experience  of  the  engineer  with  the  same  type  of  circuits,  as 
well  as  engineering  analysis  of  the  test  program  to  ensure  that  adequate 
GO/NOGO  chain  testing  is  performed. 
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Expert  System  Background 


The  development  of  this  expert  system  is  being  accomplished  in  three 
phases.  During  Phase  I,  which  has  been  completed,  a  prototype  expert 
system  (containing  the  basic  constructs  required  for  a  production  system) 
was  successfully  demonstrated  in  order  to  show  proof  of  concept.  In  Phase 
II,  a  production  expert  system  will  be  completed  which  addresses  all 
ARDEC  UUT  testing  requirements.  In  Phase  III,  the  expert  system  will  be 
expanded  to  address  the  requirements  of  test  programs  for  US  Navy  and 
Air  Force  applications. 


PHASE  I 


In  the  initial  development  phase,  the  expert  system  was  modeled  and 
implemented  as  a  prototype  capable  of  executing  all  functions  of  the 
production  expert  system,  but  on  a  smaller  scale  to  demonstrate  the 
feasibility  of  the  approach.  A  large  prototype,  the  expert  system  consists 
of  4  frames,  131  rules,  3  meta-rules,  279  parameters,  and  175  functions. 
The  development  and  execution  environment  was  a  Texas  Instruments 
Business  Pro  utilizing  the  TI  Personal  Consultant  Plus  expert  system  shell. 

Briefly  defining  the  terms  used  here  in  describing  the  prototype  expert 
system,  the  following  applies: 

1.  FRAME  -  Frames  are  used  to  break  the  domain's  knowledge  into 
smaller  chunks  of  knowledge  or  break  the  problem  into  sub-problems. 
Data  structures  such  as  rules  and  parameters  are  used  to  represent 
knowledge.  This  definition  only  applies  to  Personal  Consultant  Plus. 
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2.  RULE  -  Rules  are  used  to  represent  the  expert’s  reasoning 
methodologies.  Rules  allow  the  system  to  make  inferences  based  upon 
known  facts.  These  facts  are  represented  as  parameters.  Additionally, 
functions  can  be  called  from  rules.  An  example  is: 

If  Resistor  =  Carbon 

Then  Valid_Failmode  =  Open 

3.  PARAMETER  -  Parameters  are  used  to  represent  facts  in  the 
domain.  The  parameters  can  be  assigned  values  within  a  rule,  within  a 
function,  or  by  the  user. 

4.  META  RULE  -  These  allow  the  system  to  reason  about  the  rules, 
containing  the  expert's  reasoning  methodologies.  For  instance  (in  English): 

If  this  UUT  was  previously  entered 
Then  do  not  ask  user  for  list  of  non-detectable  faults 

5.  FUNCTIONS  -  Functions  are  subprograms  written  in  PC-Scheme, 
Texas  Instrument's  version  of  LISP  for  a  PC. 

Four  frames,  called  FAULT-SELECT,  UUT,  CIRCUIT-APP  and  TEST-PROG 
perform  individual  functions  which  relate  to  the  unique  areas  addressed 
by  the  expert  system.  FAULT-SELECT  is  the  root  frame  and  contains  45 
rules,  3  meta-rules,  142  parameters  and  175  functions.  The  purpose  of 
FAULT-SELECT  is  to  control  the  processing  of  the  expert  system  and  to 
select  the  fault  list  for  insertion  into  the  UUT  during  test  demonstration. 
UUT  is  the  child  frame  of  FAULT-SELECT  and  contains  22  rules  and  30 
parameters,  and  its  purpose  is  to  acquire  data  regarding  UUT  architecture. 
CIRCUIT-APP  is  also  a  child  frame  of  FAULT-SELECT  and  contains  40  rules 
and  77  parameters.  The  purpose  of  CIRCUIT-APP  is  to  determine  those 
faults  which  are  non-detectable  and  eliminate  those  from  the  fault  list  of 
possible  faults.  TEST-PROG  is  the  child  frame  of  CIRCUIT-APP  and  contains 
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25  rules  and  30  parameters.  Its  purpose  is  to  acquire  data  and  make 
conclusions  regarding  test  program  characteristics.  The  relationships  and 
communications  between  frames  are  illustrated  in  Figure  1. 


Scope  of  Prototype  Expert  System 


The  prototype  expert  system  was  designed  to  analyze  test  programs 
and  analog  shop  replaceable  units  (SRU)  containing  the  following 
components: 

1.  Resistors  (Carbon,  Wirewound,  Flatpack) 

2.  Capacitors  (Ceramic,  Electrolytic) 

3.  Inductors 

4.  Relays 

5.  Diodes 

6.  Transistors 

7.  Integrated  Circuits 

For  all  resistors,  capacitors,  inductors,  and  relays,  failure  mode  and 
default  failure  rate  information  is  represented  as  a  parameter  value. 


Electronic  Component  Libraries 


Diode,  transistor  and  integrated  circuit  data  is  contained  in  libraries 
accessed  by  frame  UUT.  All  library  entries  are  listed  by  commercial/Mil 
part  number.  Diodes  and  transistors  are  also  represented  in  addition  by 
default  failure  rate  values  (calculated  based  on  M1L-HBK-217  techniques) 
and  diode  or  transistor  type  used  to  assist  in  determining  the  appropriate 
fault  mode  to  be  included  in  the  fault  list  for  insertion  during 
demonstration.  Integrated  circuit  descriptions  contain,  in  addition  to  part 
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number  and  default  failure  rates,  individual  pin-by-pin  functional 
descriptions  consisting  of  logic  family,  failure  mode,  function,  and  other 
information  relevant  to  a  particular  IC. 


Domain  Expert  Interviews 


Experts  in  TPS  development  and  analysis  (the  domain)  were 
interviewed  and  general  rules  were  formulated  based  upon  the  methods 
and  procedures  used  in  analyzing  sample  test  cases  they  reviewed.  These 
rules  were  incorporated  into  the  several  frames  as  appropriate  for  the 
type  of  knowledge  involved.  For  example,  rules  regarding  test  program 
analysis  were  incorporated  into  frame  TEST-PROG,  UUT  analysis  into  frame 
UUT,  and  circuit  analysis  into  frame  CIRCUIT-APP. 


Data  Acquisition 


Data  acquisition  by  the  prototype  expert  system  was  performed 
interactively  with  the  user.  Because  the  intent  of  Phase  I  was  to  show 
concept  feasibility,  the  majority  of  efforts  were  undertaken  to  achieve  this, 
rather  tfan  to  promote  user  friendliness  and  utility.  Future  development 
will  focus  on  improving  the  user  friendliness  of  the  tool,  as  well  as 
expanding  its  capability. 


Testing 


The  ability  of  the  expert  system  to  analyze  UUT  and  test  program 
information  and  apply  expert  knowledge  to  derive  conclusions  leading  to  a 
fault  list  for  insertion  during  test  program  demonstration  was  tested 
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utilizing  three  analog  SRU  test  programs  supplied  by  ARDEC.  The  tests 
were  structured  so  that  three  human  experts  would  review  the  test 
programs  and  UUT  source  data  and  generate  fault  insertion  lists.  The 
expert  system  would  do  the  same  task  independent  of  human  experts.  The 

goal  of  a  successful  test  would  be  that  the  majority  of  human  experts 
would  agree  that  the  expert  system  obtained  valid  results  in  selecting 
components  and  failure  modes  for  insertion  at  demonstration.  It  was  not 
anticipated  that  the  expert  system  would  match  the  results  of  any  one 
human  expert,  just  as  it  is  not  anticipated  that  human  experts  will  agree 
with  each  other  in  circumstances  where  subjective  thought  is  required. 

The  results  of  testing  indicated  that  the  expert  system  selected  faults 
that  were  distributed  across  all  functional  areas  of  the  UUT  and  tested 
program  paths  throughout  the  test  program.  Of  fault  candidates  selected 
by  the  expert  system,  several  were  contained  within  the  same  ambiguity 
groups  selected  by  a  human  expert. 


Phase  I  Summary 


Summarizing  the  results  of  the  prototype  expert  system  development 
effort,  the  performance  of  the  expert  system  was  comparable  to  that  of  a 
human  expert  within  the  limited  development  domain.  Faults  selected  by 
the  expert  system  were  distributed  throughout  the  test  programs  and 
across  UUT  functional  areas,  and  the  expert  system  followed  the  same 
general  strategy  utilized  by  humans  in  performing  this  task.  All  goals  set 
for  its  performance  were  met  or  exceeded.  The  prototype  would  not  be 
satisfactory  for  general  use  as  greater  hardware  and  software  capability, 
automated  data  acquisition  and  a  high  degree  of  user  friendliness  and 
graphics  capability  would  be  required.  These  features  will  be  incorporated 
into  the  phase  II  expert  system. 
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PHASE  II 


Following  the  de\dopment  of  the  prototype  expert  system,  the  full 
scale  production  expert  system  will  fully  address,  anaiyze  and  test  analog 
shop  replaceable  unit  (SRU)  test  programs,  and  contain  provisions  for 
testing  SRU  test  programs  generated  with  the  aid  of  Automatic  Test 
Program  Generation  (ATPG). 


Expert  Svstem  Enhancements 


The  phase  II  expert  system  will  possess  significantly  greater  capability 
than  the  prototype  expert  system  developed  in  phase  I.  The  greatest 
enhancements  involve  the  automation  of  test  program  and  UUT  data 
collection  activities.  Additional  capabilities  include  the  representation  of 
17  venations  of  13  basic  electronic  component  types,  of  which  the 
following  have  been  added: 

1.  Variable  Resistor 

2.  Switch 

3.  Delay  Line  Transformer 

4.  Motor 

5.  Fuse 

6.  Circuit  Breaker 


The  phase  II  expert  system  will  vastly  expand  the  procedures  used  for 
test  program  and  UUT  analysis.  All  electronic  components  will  be  fully 
modeled,  and  component  libraries  will  be  expanded  to  include  hundreds  of 
additional  part  number  references.  One  additional  library  is  included  for 


relays.  Additional  functions  to  improve  the  utility  of  the  expert  system 
will  also  be  added. 

The  production  expert  system  will  reside  on  a  SUN  3\60  and  will  be 
implemented  in  C  language  with  the  aid  of  the  expert  system  shell 
NEXPERT  and  the  ORACLE  database  management  system.  Expert 
knowledge  will  be  implemented  in  the  form  of  rules  and  schemas.  Pattern 
matching  will  be  used  to  invoke  rules,  and  viewpoints  will  be 
implemented.  Graphics  capability  will  be  widely  implemented  to  improve 
user  interfaces.  Figure  2  illustrates  the  Phase  II  expert  system. 


Test  Program  Independent  Verification  and  Validation 


Test  program  IV&V  will  be  conducted  in  two  stages.  In  the  first  stage, 
quality  assurance  functions  are  performed  in  the  test  program  prior  to 
selecting  faults  for  inclusion  in  a  fault  list.  In  the  second  stage,  data 
regarding  UUT  architecture  and  test  program  content  are  obtained  by  the 
expert  system  through  automated  means  and  interactive  user  interface. 


Routine  Data  Analysis 


Routine  data  checks  are  conducted  in  two  main  areas:  test  program 
continuity  and  test  program/UUT  discrepancies.  The  test  program  is 
initially  checked  to  determine  that  logic  continuity  exists  throughout  the 
program.  Following  this,  the  data  content  of  the  test  program  and  the  UUT 
knowledge  base  is  compared  tor  discrepancies.  This  will  determine 
whether  the  test  program  contains  all  non-detectable  component  reference 
designations  within  ambiguity  groups,  and  whether  component  reference 
designations  exist  that  are  not  present  on  the  UUT. 
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Optional  Analyses 


Additional  analysis  of  test  program  features  may  be  performed  on  an 
optional  basis.  Contract  compliance  may  be  checked  with  regard  to 
ambiguity  group  size  percentages  as  compared  to  the  overall  program. 
Measurement  parameters  for  this  feature  may  be  adjusted  to  match 
individual  contract  requirements.  Typical  results  obtained  by  performing 
this  optional  check  are  illustrated  in  Table  1. 

Another  analysis  of  the  test  program  will  determine  whether 
components  contained  in  ambiguity  groups  are  ordered  by  failure  rate. 

This  is  accomplished  by  referencing  known  failure  rate  data,  or  default 
failure  rate  information  (contained  within  the  knowledge  base  and 
electronic  component  libraries). 

A  relative  measure  of  test  program  efficiency  may  be  obtained  based 
on  the  analysis  of  high  failure  rate  component  failure  mode  locations  in  the 
test  program.  When  high  failure  rate  modes  are  detected  early  in  the  test 
program,  a  higher  test  program  efficiency  rating  will  be  given  than  if  high 
rate  of  failure  component  detection  is  delayed  until  the  later  portions  of 
the  program. 

An  option  to  print  the  diagnostic  flow  chart  representative  of  test 
program  data  contained  within  the  knowledge  base  will  also  be  provided. 

Routine  analysis  will  be  performed  immediately  before  the  initiation  of 
the  expert  system.  Optional  analyses  are  not  anticipated  to  be  performed 
until  the  test  program  successfully  completes  all  routine  analyses. 
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Expert  System  Utilities 


Utilities  will  be  incorporated  into  the  expert  system  to  accomplish  test 
program  and  UUT  data  acquisition  both  automatically  and  through  user 
interaction.  The  ATLAS  language  test  program  will  be  parsed  to  obtain 
logic  flow  and  ambiguity  group  data.  This  will  be  accomplished 
automatically.  Data  obtained  provides  explicit  information  regarding  the 
test  program,  and  implicit  information  regarding  UUT  architecture  as 
related  to  component  makeup  and  functional  configuration.  Additional 
UUT  data  will  be  obtained  from  the  user  through  specifically  directed 
questions  and  graphics-enhanced  interactive  processing. 


Description  of  Expert  System 


From  a  list  of  approximately  100  faults,  the  expert  system  will  select 
the  best  faults  to  implement,  based  on  the  results  of  test  program  and  UUT 
analysis  and  the  application  of  expert  knowledge.  Information  associated 
with  each  fault  will  be  grouped  together  as  a  single  entity  for  each  fault. 
The  data  structure  used  to  represent  this  single  entity  is  referred  to  as  a 
schema. 

Schemas  are  used  to  describe  objects  or  events.  An  object  has 
associated  characteristics  and  these  are  represented  as  slots  of  the  schema. 
The  actions  typically  associated  with  an  event  would  be  represented  in  the 
slots  of  a  schema  describing  an  event.  Schemas  can  also  be  grouped 
together  to  describe  more  complex  objects  or  events. 

The  expert's  reasoning  strategies  will  be  incorporated  into  forward  and 
backward  chaining  rules.  A  forward  chaining  rule  starts  with  some  initial 
data  and  works  forward  to  make  conclusions  based  upon  the  data.  A 
backward  chaining  rule  begins  by  assuming  a  conclusion/hypothesis  is 
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true  and  tries  to  prove  the  conditions  necessary  for  this  hypothesis.  These 
two  strategies  will  be  combined  to  try  to  emulate  the  expert’s  reasoning 
process  as  closely  as  possible. 

The  forward  chaining  and  backward  chaining  processes  may  be 
interrupted  by  a  message  from  the  user,  another  process  or  function,  in 
order  to  process  a  goal.  These  goals  will  have  a  priority  level  associated 
with  them.  Additionally,  during  the  problem  solving  process  alternate 
goals  can  be  generated  dynamically.  For  example,  when  the  current 
strategy  appears  to  fail,  an  alternate  strategy  can  be  pursued.  Acceptance 
test  personnel  can  also  employ  what-if  hypothetical  reasoning  to  explore 
changes  in  fault  selection  when  the  emphasis  is  shifted  from  failure  rate  to 
specific  UUT  functional  areas,  or  to  particular  test  program  paths. 

The  expert  system  is  based  upon  non-monotonic  reasoning,  which  will 
cause  the  system  to  reevaluate  past  inferences  when  new  information  is 
available.  New  information  may  include  resetting  or  changing  a  hypothesis 
or  a  subgoal,  caused  by  exploring  hypothetical  situations.  Or  a  goal  may  be 
reset  during  an  iteration  process.  It  is  also  possible  that  this  new 
information  will  cause  a  contradiction  in  the  knowledge  base.  The  system 
will  detect  this  contradiction  and,  according  to  the  setting  of  a  switch, 
either  change  its  current  hypothesis  or  ignore  the  new  information.  This 
last  step  is  similar  to  the  way  humans  alter  their  knowledge  when  new 
information  becomes  available.  Where  operations  which  cannot  be 
performed  within  the  context  of  the  expert  system  development  tool  are 
required,  special  functions  written  directly  in  C  will  be  developed. 

In  order  for  the  engineer  to  understand  the  reasoning  process  behind 
the  system's  fault  selection,  two  operations  are  provided.  One  provides  the 
user  with  the  reasoning  behind  the  current  strategy.  The  other  gives  the 
complete  line  of  reasoning  used  in  extracting  the  reasoning  processes  used 
by  experts,  including  asking  the  engineer  to  select  the  faults  with  a  time 
constraint  or  providing  the  engineer  with  incomplete  data  to  make  his/her 
selection.  The  knowledge  acquisition  process  will  continue  throughout  the 
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development  of  the  expert  system.  To  ensure  the  validity  of  the  expert 
system’s  reasoning  process,  the  engineers  will  periodically  be  provided 
with  the  faults  selected  by  the  expert  system  and  the  reasoning  used  in 
making  these  selections. 


PHASE  III 


A  third  phase  of  development  is  anticipated  following  completion  of  the 
full  scale  production  expert  system  for  ARDEC.  Further  enhancements  will 
be  implemented  with  regards  to  digital  SRU  and  LRU  test  program  analysis 
capability,  and  in  addition  to  research  in  more  sophisticated  means  of 
knowledge  acquisition  for  expert  rules.  In  addition,  direct  UUT  data 
acquisition  through  optical  schematic  scanning  and  other  techniques  will 
be  investigated.  Presently,  more  than  7  potential  users  in  the  Army,  Navy 
and  Air  Force  have  expressed  interest  in  Phase  III  participation.  Many  of 
the  innovations  being  developed  in  this  project  have  applications  in  the 
commercial  realm  as  well,  particularly  in  the  area  of  automated  knowledge 
acquisition  and  pattern  recognition. 


SUMMARY 


The  concept  of  utilizing  an  expert  system  for  evaluation  of  test 
programs  has  been  successfully  demonstrated  for  analog  shop  replaceable 
units.  The  full  scale  development  of  a  UNIX  based  expert  system  utilizing 
automated  test  program  and  Unit  Under  Test  (UUT)  data  collection,  will 
result  in  a  practical,  user  friendly  tool  capable  of  augmenting  a  QA/TPS 
acceptance  engineer  during  independent  verification  and  validation.  The 
expert  system  will  provide  the  QA  engineer  with  previously  unavailable 
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information,  results  of  analyses  and  access  to  human  expert  knowledge 
regarding  test  program  and  UUT  testing,  resulting  in  greater  human 
efficiency  in  performing  this  task. 


Table  1.  Ambiguity  Group  Size  Percentages 


AMBIGUITY 

GROUP  SIZE 

PERCENTAGE 

5  or  less 

90% 

6  to  8 

6% 

9  to  11 

0% 

12  or  more 

4% 

15 


17 
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Figure  2  -  Expert  System  for  TPS  Verification/Validation 
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